Customization of help information based on eeg data

ABSTRACT

A method implemented by a computing device for helping a particular user use a user interface (UI). Electroencephalography (EEG) data is obtained that indicates brain activity of a particular user during a period in which that user views the UI and/or interprets help information that describes how to use the UI. Based on the EEG data, the computing device selects, from among multiple predefined cognitive states, the one or more cognitive states that characterize the particular user during the period. The computing device assists the particular user to use the UI by customizing the help information for the particular user based on the one or more selected cognitive states. A complementary computing device and computer program product are also disclosed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/574,911, filed on Jan. 13, 2022 (status pending), which is acontinuation of U.S. application Ser. No. 17/021,462, filed on Sep. 15,2020 (now U.S. Pat. No. 11,238,748 issued on Feb. 1, 2022), which is acontinuation of U.S. application Ser. No. 15/521,143, filed on Apr. 21,2017 (now U.S. Pat. No. 10,810,896, issued on Oct. 20, 2020), which isthe National Stage Entry of International Patent Application No.PCT/SE2014/051252, filed on Oct. 24, 2014. The above identifiedapplications and patents are incorporated by this reference.

TECHNICAL FIELD

This application relates to customizing help information (e.g.,customizing help information based on electroencephalography data).

BACKGROUND

Electroencephalography (EEG) is the measurement of electrical activityin a person's brain as measured on a person's scalp. The electricalactivity is derived from ionic current flow within the brain's neurons,and can therefore serve as an indication of activity within the brain.By localizing measurements on a subject's scalp, the EEG data can beidentified in various brain locations supporting mental processing.Because different areas of the brain are associated with different typesof processing, EEG can provide an indication as to the cognitive stateand cognitive activity of the person being monitored.

Event-related potentials (ERPs) can be determined from EEG data tomeasure responses to specific events. A variety of ERPs are known tothose of ordinary skill in the art. These include P300, N200, P200, etc.The “P” or “N” preceding an ERP name refers to whether it is a positiveor negative measured potential, and the numeric value (e.g., 200, 300)typically refers to how long after a stimulus the measurement occurred.An example stimulus could include a user viewing a specific image orhearing a sound, for example. FIGS. 1 and 2 include graphs 10, 14 thatillustrate example brain potentials 12, 16 and example ERPs of thosemeasured potentials.

The P300 ERP, for example, is a positive EEG signal occurringapproximately 300 ms after a given stimulus. The P300 ERP is related tothe categorization of an event, and is stronger when an event isrecognized. It can therefore be used to determine whether or not astimulus is known to a user. It can be divided into “P3a” and “P3b”,where P3a relates to more rare instances (i.e., where more effort isrequired to recollect the stimulus), and P3b relates to stimuli that areeasier to recollect.

The N200 ERP is a negative EEG signal occurring approximately 200 msafter a given stimulus, and can be associated with stimulusidentification, attention, and inhibition of motor responses.

The P200 ERP (also referred to as “P2”) is a positive EEG signaloccurring approximately 200 ms after a given stimulus. It is related,among other things, to the processing of language. It can be used todetect, for example, if a word that is not expected by a user appears ina phrase.

EEG data may also include measurements of neural oscillations. Based onthe frequency of neural oscillations, a type of brain wave can bedetected. For example, theta waves have a frequency range of 4-7 Hz,alpha waves have a frequency range of 7.5-12.5 Hz, and beta waves have afrequency range of 12.5-30 Hz. Neural oscillations have been linked tocognitive states, such as awareness and consciousness. FIG. 3 is a graph18 showing example neural oscillations 20 for an alpha wave.

EEG data can be measured using a so-called “brain computer interface”(BCI) which measures brain activity and communicates those measurementsto a computing device. BCIs have historically required a user to wear acap containing many EEG sensors. More recently, a headset style of BCIhas become available through companies such as Emotiv that has a smallerprofile and a more stylized appearance.

SUMMARY

According to one aspect of the present disclosure, a method of helping aparticular user use a user interface (UI) is implemented by a computingdevice. The computing device obtains electroencephalography (EEG) datathat indicates brain activity of a particular user during a period inwhich that user views the UI and/or interprets help information thatdescribes how to use the UI. The EEG data may be obtained from abrain-computer interface (BCI), for example. Based on the EEG data, thecomputing device selects, from among multiple predefined cognitivestates, the one or more cognitive states that characterize theparticular user during the period. The computing device assists theparticular user to use the UI by customizing the help information forthe particular user based on the one or more selected cognitive states.

In some embodiments, the method is further characterized by thecomputing device selecting, based on the EEG data and from amongmultiple predefined emotional states, the one or more emotional statesthat characterize the particular user during the period. In suchembodiments, the customizing of the help information for the particularuser is further based on the one or more selected emotional states.

In some embodiments, the EEG data includes Event Related Potentials(ERPs) corresponding to individual elements of the UI that the userviews and/or corresponding to specific words of the help information theuser interprets. In such embodiments, selecting the one or morecognitive states includes comparing the ERPs to one or more ERPthresholds, and determining, from rules that define the cognitive statesas a function of said comparing, which of the predefined cognitivestates are indicated by the determined ERPs.

In some embodiments, the EEG data includes neural oscillations of theparticular user during the period. In such embodiments, selecting theone or more cognitive states includes determining a neural oscillationtype based on a frequency of the neural oscillations, comparing anamplitude of the neural oscillations to a neural oscillation thresholdfor the neural oscillation type, and determining, based on the comparingof the amplitude to the threshold, which of the predefined cognitivestates are indicated by the frequency and the amplitude of the neuraloscillations.

Gaze tracking data may be used to determine one or more individualelements of the UI to which the one or more predefined cognitive statescorrespond, and/or to determine one or more individual words of helpinformation to which the one or more predefined cognitive statescorrespond.

The customizing of help information may include any combination of thefollowing, for example: customizing an organization of help information;adding or removing a definition of a word in a section of helpinformation; increasing or decreasing a level of detail in a section ofhelp information; increasing or decreasing a level of complexity of thelanguage used in a section of help information; adding or removingreferences to a previous version of the UI in a section of helpinformation; adding or removing a shortcut to help information; andadding a brief snippet of help information, such as a tooltip, to theUI.

In some embodiments, the UI is a graphical user interface of a softwareprogram that is accessed using the computing device, and the helpinformation provides instruction on how to use the software program. Insome embodiments, the UI is a UI of a machine, and the help informationprovides information on how to use the machine.

According to one aspect of the present disclosure, a computing devicefor helping a particular user use a UI is disclosed. The computingdevice is configured to obtain EEG data that indicates brain activity ofa particular user during a period in which that user views a UI and/orinterprets help information that describes how to use the UI. Thecomputing device is further configured to, based on the EEG data andfrom among multiple predefined cognitive states, select the one or morecognitive states that characterize the particular user during theperiod. The computing device is further configured to assist theparticular user to use the UI by customizing the help information forthe particular user based on the one or more selected cognitive states.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1-2 illustrate some examples of known ERPs.

FIG. 3 illustrates example neural oscillations of an alpha wave.

FIG. 4 illustrates an example system for using EEG data to customizehelp information.

FIG. 5 illustrates another example system for using EEG data tocustomize help information.

FIG. 6 schematically illustrates an example method of helping aparticular user use a user interface (UI).

FIG. 7 illustrates an example user interface.

FIG. 8 schematically illustrates an example computing device configuredto customize help information based on EEG data.

DETAILED DESCRIPTION

A user's understanding of the functionality of a piece of software or adevice changes from user to user. Despite this, in the prior art helpinformation for software and devices has been the same for all users.Also, help information provided with software has not changed over timeto reflect a user's change in understanding of the software. The presentdisclosure provides a way for the understanding of an individual user tobe assessed, and for that information used to create a help information(e.g., in the form of a help page) that is personalized to that user'slevel of understanding. Help information may include any combination oftext based information, audio, and video, for example.

A method, apparatus and computer program product for helping aparticular user use a UI based on EEG data are disclosed. EEG data isobtained by measuring electrical potentials that indicate brain activityof a particular user during a period in which that user views a UIand/or interprets help information that describes how to use the UI.Based on the EEG data and from among multiple predefined cognitivestates, a selection is made of the one or more cognitive states thatcharacterize the particular user during the period. The particular useris assisted to use the UI by customizing the help information for theparticular user based on the one or more selected cognitive states.

In some embodiments, the cognitive states are tied to cognitiveprocesses, such as memory, association, language, attention, patternrecognition, problem solving, etc. Because of this, the cognitive statecan be used to determine a particular user's familiarity orunderstanding of some functionality offered by software being used. Thecognitive state can relate to the user viewing a UI and/or interpretinghelp information (e.g., reading help information, listening to audiohelp information, or listening or watching video help information). Insuch embodiments, the cognitive states may characterize a state of oneof more of the cognitive processes.

Gaze tracking is a technique whereby the location of a person's gaze(for example which area on a computer screen the user is looking at) ismonitored. Various systems are available that perform gaze tracking, themost common being the use of a camera to monitor movement of the eye,which together with an understanding of the local environment (e.g., thedistance to a screen that is being looked at, relative position of thecamera to the screen, etc.) is used to assess the area, e.g., a portionof the UI or an element of the UI, the user is looking at. Gaze trackingmay optionally be used in combination with the EEG data described above,such that the obtained EEG data is correlated to specific UI element(s)and/or individual words of help information to which the cognitive statecorresponds. By using gaze tracking, the customization can be based onthe specific UI element(s) and/or individual words identified. If gazetracking data is used, an operating system of the computing devicecustomizing the help information (or an operating system of a secondarycomputing device performing gaze tracking on behalf of the computingdevice customizing the help information) needs to know the relativelocation of the physical objects relative to the camera that tracks theuser's gaze.

By analyzing EEG data, a computing device providing help information canestimate a person's understanding of the activity they are undertakingon a device (e.g., modifying settings of a software application, anoperating system, or the computing device). The estimation ofunderstanding is made via signals measured using a BCI apparatus capableof measuring electrical potentials which represent electrical activityat one or more locations on the user's scalp. The signals from the EEGapparatus can be used to identify instances of degraded understanding ofthe activity they are undertaking on a device.

A computing device can, in some embodiments, estimate whether adegradation of understanding relates to: the software application beingused (i.e., is the user familiar with the application or software beingused); a comparison between functions within the software applicationprogram being used (e.g., is the user seemingly unfamiliar with aportion of the functionality of the software application being used,while being seemingly familiar with other aspects of the software); anda comparison to functions provided by other software applications (e.g.,is the user familiar with similar functionality in other softwareapplications, while having a degraded familiarity or understanding withthe functionality within the current software application).

The user's familiarity or understanding of software functionality may bebased on one or more of: whether a UI element is recognized, whether aUI element or help information contains words that are not expected bythe user, and whether the user's cognitive load appears higher whenattempting to use a piece of functionality in comparison to other piecesof functionality offered by the application software.

In accordance with the user's estimated familiarity or understanding ofa function, different types of help may be offered. Examples of how thehelp can be modified include: the content can be modified (e.g., thecomplexity of language can be changed, or the number and simplicity ofexamples can be adjusted); the order of content can be modified (e.g.,for users with lower familiarity the help text can provide a generaldescription followed by specific functionality, whereas for users withhigher estimated levels of familiarity the help text can invert thisorder by providing the specific functionality first followed by thegeneral description); additional contextual information can be provided(e.g., if a user is seemingly familiar with a similar piece offunctionality in a different software application, but seeminglyunfamiliar with the current application, then additional contextualinformation can be provided as to the significance of the functionalityin the current application); and if the user appears to have a moredistant recollection with regard to a piece of functionality then thehelp can be modified to describe how the user previously utilized thatfunctionality.

Thus, according to one aspect of the present disclosure, a particularuser's brain activity is measured using an EEG apparatus (e.g., a BCI)while the particular user operates a device via its UI. The UI may be agraphical user interface (GUI) delivered via a screen, or physical inthe form of objects (e.g., buttons, dials, etc.). The user's gaze may besimultaneously tracked such that the UI elements being viewed at aspecific time are known, and such that the EEG measurements can beassociated with individual UI elements. The EEG measurements are thenused to estimate the user's familiarity and/or understanding of thefunctionality associated with the UI element. Based on this estimationof familiarity and/or understanding, any help that is accessed by theparticular user is tailored in such a way to have a greater likelihoodof being appropriate to the particular user's needs.

FIG. 4 illustrates an example system 30 for using EEG data to customizehelp information in which the UI is a graphical user interface (GUI) ofa software program that is accessed using computing device 36, and thehelp information provides instruction on how to use the softwareprogram. The system 30 includes a brain computer interface (BCI) 32operative to record EEG data from a user 34, and includes the computingdevice 36 which is operative to provide help information to the user 34(e.g., via its electronic display and/or via speakers). A camera 38 mayoptionally be included to perform gaze tracking of the user 34 forcorrelating the EEG data with specific items viewed by the user 34 (seedotted lines indicating gaze tracking). Although the BCI is shown as aheadset 32, it is understood that other BCIs may be used (e.g., a scalpcover that includes a plurality of electrodes). Also, although thecomputing device 36 is shown as a tablet, it is understood that othercomputing devices could be used, such as laptop computers, desktopcomputers, smartphones, etc.

FIG. 5 illustrates an example system 40 for using EEG data to customizehelp information in which the UI is a UI of a machine 42, and the helpinformation provides information on how to use the machine 42. Thesystem 40 includes the machine 42, a BCI 32 operative to record EEG datafrom a user 34, and also includes a computing device 44 associated withthe machine 42. The computing device 44 is configured to provide helpinformation for the machine 42 (e.g., via its electronic display and/orvia speakers). The machine includes a number of controls, including aplurality of buttons 46, knobs 47, dials 48, sliders 49, and switches50, and may optionally also include a camera 52 to perform gaze trackingof the user 34 for correlating the EEG data with specific items viewedby the user 34 (see dotted lines indicating gaze tracking). In FIG. 5 ,the UI elements at issue are physical objects, such as physical buttonson a device. It will be appreciated that an alternative environment tothe industrial machinery of FIG. 5 could be the interior physicalcontrols of a car.

FIG. 6 illustrates an example method 100 of helping a particular useruse a UI. EEG data is obtained (block 102) that indicates brain activityof a particular user during a period in which that user views the UIand/or interprets help information that describes how to use the UI.Based on the EEG data a selection is made (block 104) from amongmultiple predefined cognitive states of the one or more cognitive statesthat characterize the particular user during the period. The particularuser is assisted (block 106) to use the UI by customizing the helpinformation for the particular user based on the one or more selectedcognitive states.

The customization of help information may also be based on emotionalstates, such as frustration, anger, happiness, excitement, etc., whichmay or may not be tied to cognitive processes. Thus, in one or moreembodiments the method 100 also includes selecting, based on the EEGdata and from among multiple predefined emotional states, the one ormore emotional states that characterize the particular user during theperiod. In such embodiments, the customization of block 106 for theparticular user is further based on the one or more selected emotionalstates.

Some example cognitive states may include a default/normal state, anaware state, an unaware state, semantic understanding, a lack ofsemantic understanding, high cognitive load with understanding, highcognitive load with a lack of understanding, attentive, low attention,and active.

Some example customizations that can be performed for help informationmay include one or more of the following: creating a customizedorganization of help information that differs from a defaultorganization of the help information; adding or removing a definition ofa word in a section of help information; increasing or decreasing alevel of detail in a section of help information; increasing ordecreasing a level of complexity of the language in a section of helpinformation; adding or removing references to a previous version of theUI in a section of help information; and adding or removing a shortcutto help information.

Of course these are only example customizations, and it is understoodthat other customizations could be performed.

In some embodiments, the EEG data obtained in block 102 includes ERPscorresponding to individual elements of the UI that the user viewsand/or corresponding to specific words of the help information the userinterprets. In such embodiments, the selecting of the one or morecognitive states (block 104) includes comparing the ERPs to one or moreERP thresholds, and determining, from rules that define the cognitivestates as a function of the comparing, which of the predefined cognitivestates are indicated by the determined ERPs. For example, the ERP P300being greater than a threshold x could indicate a first cognitive state,and being less than the threshold x could indicate a different secondcognitive state. Some example rules are shown in Table 1, which isdiscussed in more detail below. Prior to being compared to the one ormore ERP thresholds, the ERP data (e.g., signals from which the ERPs arecalculated) may be processed in some way (e.g., any combination offiltering, normalizing, value averaging, etc.). Thus, when used herein,“ERP data” may refer to a processed ERP signal, an ERP signal that omitssuch processing, or a combination thereof.

Determining the ERPs may be based on specific items viewed by theparticular user. In some embodiments, obtaining EEG data that indicatesbrain activity of the particular user (block 102) includes: obtainingEEG measurements from a BCI configured to measure EEG data from theparticular user's brain during the period, and determining the ERPsbased on the EEG measurements and based on times at which the user viewsthe individual elements of the UI and/or times at which the userinterprets the help information that describes how to use the UI.

The times at which the user views the individual elements of the UIand/or interprets help information can be determined using gaze trackingdata, for example.

Neural oscillations (NOs) may be used as the basis for the helpinformation customizations, either alone, or in combination with ERPs.Thus, in some embodiments, the EEG data includes neural oscillations ofthe particular user during the period. In such embodiments, selectingthe one or more cognitive states (block 104) includes the following:determining a neural oscillation type based on a frequency of the neuraloscillations; comparing an amplitude of the neural oscillations to aneural oscillation threshold for the neural oscillation type; anddetermining, based on the comparing of the amplitude to the threshold,which of the predefined cognitive states are indicated by the frequencyand the amplitude of the neural oscillations.

In some such embodiments, obtaining EEG data that indicates brainactivity of the particular user (block 102) includes obtaining EEGmeasurements from a BCI configured to measure EEG data from the user'sbrain during the period, and determining, from the EEG measurements, thefrequency and amplitude of neural oscillations of the user during theperiod. Thus, the EEG data may include a set of frequencies andamplitudes of neural oscillations, or may include the wave formsrepresenting the neural oscillations such that the computing deviceperforming method 100 needs to determine the frequencies and amplitudesitself from the waveforms.

In embodiments in which gaze tracking data is used, the customization ofhelp information may be based on specific UI elements viewed and/orspecific sections of help information viewed. The section of helpinformation to which EEG data is correlated could vary in size (e.g.,several paragraphs on a topic, or several words from a single sentence).For example, a customization could be performed based on a user beingfrustrated while reading several paragraphs of help information. Asanother example, a customization could be performed based on a userhaving a semantic misunderstanding within a phrase or sentence that ismuch shorter.

In some embodiments in which gaze tracking data is used, obtaining theEEG data (block 106) includes determining the EEG data based on gazetracking data that indicates which individual elements of the UI theuser views during the period. In some of these embodiments, the method100 also includes determining, based on the EEG data, one or moreindividual elements of the UI to which the one or more predefinedcognitive states correspond. The customizing of block 106 for theparticular user is then further based on the one or more individualelements of the UI to which the one or more predefined cognitive statescorrespond.

In some embodiments where gaze tracking data is used, obtaining the EEGdata (block 106) includes determining the EEG data based on gazetracking data that indicates which help information the user viewsduring the period. In such embodiments, the method 100 also includesdetermining, based on the EEG data, one or more individual words of thehelp information to which the one or more predefined cognitive statescorrespond. The customizing of block 106 for the particular user is thenfurther based on the one or more individual words of the helpinformation to which the one or more predefined cognitive statescorrespond.

Table 1, shown below, lists example cognitive states, descriptions ofthose cognitive states, and conditions that can cause the cognitivestates. The conditions can refer to the amplitude of various ERPs, orthe amplitude of various neural oscillations, for example. Table 2presents similar information, but for emotional states. Table 3,meanwhile, lists example customizations that can be performed for agiven cognitive or emotional state.

TABLE 1 Example Cognitive State Lookup Table Condition Event CognitiveState Description Baseline ERP and Default/normal state User is notconfused neural oscillations and understands UI items they are viewingP300 < x Viewing image Aware/Understanding Use is aware of a visualstimuli P300 > x Viewing image Not aware/ Use is not aware ofMisunderstanding and/or misunderstands a visual stimuli N400 > y Viewingword in Semantic understanding Word makes sense in sequence of textcontext N400 < y Viewing word in lack of semantic Word does not makesequence of text understanding sense in context P300 < x and Viewingimage and Aware + Lack of Use is aware of a N400 < y text semanticunderstanding visual stimuli but does not understand text P300 > x andViewing image and Not aware + lack of Use is not aware of a N200 < ytext semantic understanding visual stimuli and does not understand textP300 > x, Viewing image High cognitive load User has to think a lot n400< y, about what they are high amplitude theta viewing, does not waves,recognize items on the high amplitude beta and/or text (lack ofunderstanding) screen, and is waves confused by the text used. Thiscould indicate confusion. P300 > x, Viewing image High cognitive loadUser has to think a lot n400 > y, and/or text (with understanding) aboutwhat they are high amplitude theta viewing. This could waves, indicatethat the user is high amplitude beta undertaking a complex waves task.High amplitude n/a Low attention Associated with being alpha wavesrelaxed, drowsiness and sleep Low amplitude n/a Attentive User isattentive to alpha waves external stimuli High amplitude beta n/a ActiveActive, busy, anxious waves thinking, active concentration. Exertingmental effort, retrieving memories.

In some embodiments, if the “default/normal state” is detected, then nocustomization is performed on help information presented to theparticular user. Of course, this is only an example, and it isunderstood that in some embodiments customization may always beperformed.

The thresholds shown in Table 1 may be user-specific thresholds that aredetermined based on baseline EEG data for the particular user and/orbaseline EEG data for one or more other users. For example, wheninitiating a software program, a user may be asked about a feature thatthey understand well, and an inference could be made that the user has agood understanding of that feature. As another example, an automaticdetection may be made of a UI element that is frequently used by theuser, and an inference could be made that the frequently used UI elementrepresents a feature that is well-understood by the user. This could beperformed automatically without “asking” the user as in the previousexample, such that the user may be unaware of what feature is being usedto determine their baseline EEG data. By having baseline EEG data, amore accurate determination can be made of the delineation betweenvarious cognitive states. EEG data from other users may also be used.For example, if some features are very commonly used by many users(e.g., a ‘Save’ button), an inference may be made that the commonly usedfeature is well-understood by most users, and based on that a predictionmay be made that the particular user will also understand that givenfeature.

Help information can also be customized based on the emotional state ofthe particular user. For example, if a given user is frustrated, angry,happy, etc., then help information provided to the particular user maybe customized accordingly. Table 2, shown below, lists an exampleemotional state, a description of the emotional state, and a conditionthat can cause the emotional state. It is understood that the use ofemotional states is optional, and that in some embodiments onlycognitive states may be used for customizing help information. Also, itis understood that other emotional states not shown below could be usedas the basis for customizing help information.

TABLE 2 Example Emotional State Lookup Table Condition Event CognitiveState Description High amplitude theta n/a Frustration Theta waves willwaves occur for brief periods during an emotional response tofrustrating events or situations

In one example, an emotional state of a user is determined based on anasymmetry of frontal lobe EEG measurements (e.g., if the right side ofthe front of the brain has a high EEG signal and the left side low, orvice versa).

Table 3, shown below, lists example cognitive and emotional states, andexample descriptions of the customizations that can be applied to helpinformation based on those states.

TABLE 3 Example Customizations Cognitive/Emotional State CustomizationAction Normal/default state No customization of help performed. Highcognitive load Organize help so that information is presented in asimple way, with (lack of understanding) a clarification step after eachsegment of new information (ii) If the user has spent time in thiscognitive state (and therefore has spent effort trying to understand),organize help such that it has a list whereby the user is asked whataspects they do or do not understand and then provide additional detailsfor the sections that are not understood (either user could state whatthey don't understand or their User State could be determined usingEEG). Not aware (looking at an Because user does not recognize imagethey are looking at, use a icon, first use of an ‘quick tip’ to providea written description of the icon, as it may just application) be thatthe user does not associate the icon with some known functionality.Aware + High cognitive User is thinking hard, and is understanding whathe/she is doing. load (understanding) Organize help so that ‘parameter’style information is presented first (e.g., a table of symbols/functionswith their meanings/parameters is presented first on the assumption thatthe user generally knows/understands the function but is searching for asmall detail). Aware (application Present the help in a way thatexplains the new functionality, but icon) + Not aware consider thereader to be someone who is has an understanding of (functionality icon)other aspects of the application in question (i.e., treat as a newfunction the user has not seen before, but in an application that theuser is familiar with). Lack of semantic Either explain what words meanor provide an alternate set of understanding words. Awareness + Lack ofFor example the user may recognize the icon but is confused by thesemantic understanding written explanation. This may be due to somechanges between versions in the application, in which case organize helpso it refers back to the previous version. Alternatively, thefunctionality may have some specific nuances that could be explained.Low attention Present help in a way more likely to grab attention, forexample an animated character could interact with the user. AttentiveThe user is interested in the topic. Provide ‘find out more’ sections atthe end of the help information. Frustration Organize help such that theuser can experience a successful interaction to reduce frustration. Forexample the user could interact with the device to perform a function,but do so in a way where the interaction is broken down into sections,allowing the user to experience several small successes as they completeeach section.

In some embodiments the customizing of help information (block 106) forthe particular user includes customizing an organization of helpinformation if the cognitive state indicates high cognitive load orawareness of an element of the UI, or if the emotional state isfrustration.

In some embodiments the customizing of help information (block 106) forthe particular user includes adding or removing a definition of a wordin a section of help information if the selected one or more cognitivestates indicate a lack of a semantic understanding.

In some embodiments if the selected one or more cognitive statesindicate a high cognitive load, the customizing of help information(block 106) for the particular user includes increasing or decreasing alevel of detail in a section of help information and/or increasing ordecreasing a level of complexity of the language used in a section ofhelp information.

In some embodiments, the selected one or more cognitive states indicatea lack of semantic understanding and/or awareness of an element of theUI, and the customizing of help information (block 106) for theparticular user includes adding or removing references to a previousversion of the UI in a section of help information. This particularcognitive state could occur if a user recognizes an icon but is confusedby the written explanation for the icon, possibly because of somechanges between versions in the application. The adding or removingreferences to a previous version of the UI could include providing theold name of the UI element and/or providing an explanation of how the UIelement worked in the previous release compared to how it works in thecurrent release being used by the particular user.

In some embodiments, the selected one or more cognitive states indicatea lack of awareness and/or a lack of semantic understanding, and thecustomizing of help information (block 106) for the particular userincludes adding or removing a shortcut to the help information and/oradding a brief snippet of help information to the UI (e.g., a so-called“quick tip” or a “tooltip” which is displayed when the user hovers apointer of a graphical UI, such as a mouse pointer, over a UI element).

Referring now to FIG. 7 , an example user interface 60 is shown whichincludes a plurality of UI elements that include settings 62A-621 for asmartphone or tablet computing device. Assume that the user is lookingat the settings starting from the bottom up, such that the setting 62Afor “Wallpapers & Brightness” is viewed first and the setting 62I for“Airplane Mode” is viewed last. Assume also that the user understandseach of the settings 62A-62H but is confused by the setting 62I for“Airplane Mode.” An ERP signature of the particular user will reflectthat they are familiar with settings 62A-H and are not familiar withsetting 62I. For example, the user's P300 response may be below athreshold x for each of settings 62A-I but may be above that thresholdfor setting 62I, indicating that the user has a lack of understanding ofthe setting 62I.

The computing device implementing method 100 can respond in severalways. In some examples, the help may be automatically provided. In oneexample, the computing device could automatically provide a “quick tip”for that feature, such that a brief snippet of text is provided toexplain the “Airplane Mode” and to serve as a link to more informationabout the airplane mode. This could be done automatically based on theEEG data without the user actually requesting help. In another example,the computing device could present an icon or other indicator next to“Airplane Mode” to indicate that help is available for “Airplane Mode”(e.g., a light bulb or some other icon appearing next to the setting 62Ifor “Airplane Mode”). In another example, no automated help ispresented, but when the particular user requests help for the interface60, the help presented for “Airplane Mode” is listed first, or isexplained in greater detail.

In the prior art, the help associated with a piece of software or deviceis typically either ON (such that tips are given, e.g., the first time apiece of functionality is used), or is provided to the user on request(e.g., when the user clicks on a “Help” button). The present disclosureprovides techniques for automatically providing help based on EEG datawhen a particular user has a cognitive state and/or emotional statedthat is commensurate with the need for help. This enables helpinformation to be personalized for an individual user and/or to be moreappropriate to the user's understanding of the software.

Referring again to FIG. 7 , as another example, assume that EEG data forthe particular user indicates that the user is generally frustrated (notnecessarily at a particular UI element, but just in general). Asdescribed in Table 2 above, this may be detectable if the particularuser has neural oscillations that are theta waves with high amplitude(i.e., amplitude above a predefined threshold). Help for the particularuser could be customized such that the user can experience a successfulinteraction to reduce frustration (see last row of Table 3 above).

In some embodiments, the method 100 includes three separate processes:A) cognitive and/or emotional state determination: determining whichspecific UI element or UI element sequence to which the user's cognitiveand/or emotional state corresponds; B) help request: receiving a helprequest whereby the user requests help; and C) PHP creation: creation ofa customized, personalized help page (PHP) that includes helpinformation customized for the particular user based on activity beingundertaken by the user and their cognitive and/or emotional state.

Each of these processes is discussed in greater detail below.

Cognitive and/or Emotional State Determination

In some embodiments, the determination of the user's cognitive and/oremotional state includes the following: i) tracking the particularuser's gaze with respect to the location of a camera; ii) an operatingsystem (OS) of the computer, or an application software being executedon the computer, implementing method 100 associating the location of theuser's gaze to a specific location on the screen, and thus to a UIareas, and thus to specific UI elements; iii) the operating system (OS),or the application software, recording the time the gaze enters andexits the UI area; and iv) measuring of electrical activity on theparticular user's scalp using a BCI.

A processing unit within the computing device implementing method 100separates a number of specific ERPs and records each of these, togetherwith the time of its occurrence. The computing device takes the EEG datasignals and determines the user's cognitive and/or emotional state.Here, the initiation of an event is defined as the time the user's gazeenters a UI area. For example, the cognitive and/or emotional state maybe determined by performing the following: i) determining that an ERP isabove or below a certain threshold stored in a lookup table (e.g., Table1 or 2 above); ii) normalizing the ERP for a first UI element withrespect to a second UI element, which is known to be well understood bythe user, and determining the normalized value is above a certain value;iii) determining whether a sequence of ERPs as the user moves between UIareas matches a pattern described in an ERP lookup table (e.g., Tables1-2 above).

Similar steps could be performed to determine what section of helpinformation was viewed by a user, and to determine ERPs accordingly. Asdiscussed above, a section of help information can vary in size (e.g.,several paragraphs on a topic, or several words from a single sentence).

Where it is beneficial to either normalize the ERP, or use a comparativeERP measurement versus a known signal (e.g., one associated withfunctionality that the user is familiar with and understands well), thencertain well understood and commonly used functions such as ‘Save’ orthose that associated with icons that are consistent across a variety ofsoftware, are designated as reference ERP signals (e.g., for determininga “baseline” for a given user). In this regard a cognitive and/oremotional state can be associated with a specific UI element or piece ofhelp information (or a sequence of UI elements or pieces of helpinformation, respectively).

Help Request

In this process, the particular user requests help. Examples of how thismight be done include: by interacting with the OS or the applicationsoftware (e.g., pressing a button or link or giving a specific voicecommand), by using a setting such that help is automatically displayedonce a specified cognitive and/or emotional state (or sequence ofstates) are detected, and/or by using a setting such that help isautomatically provided once an ERP, or normalized ERP, reaches aspecified threshold.

As used herein, “normalization” can refer to any of a number of standardmathematical forms (e.g., standard deviation, standard score, mean,etc.). For example, normalization could refer to adjusting two valuesmeasured on different scales to a common scale. Thus, if a measurement“A” on a scale of 0 to 20 was 4, and a measurement “B” on a scale of 0to 50 was 5, then normalizing the two sets to a scale of 0 to 1 wouldmean A(normalized)=0.2 and B(normalized)=0.1.

PHP Creation and Display

Once help has been requested and the cognitive and/or emotional statesfor a given set of UI elements and/or sections of help information areknown, customized help information may be generated. The customized helpinformation could be presented in the form of a personalized help page(PHP), for example. Alternatively, the customized help information couldbe provided in other ways (e.g., on other forms than a page, such asaudio playback while the user continues viewing the UI). An example ofhow customized help information may be created can include thefollowing.

Sections of help information may be organized as “help segments,” eachbeing (i) categorized by function, and where appropriate (ii) associatedwith a given UI element and (iii) ranked or tagged using a set ofInformation Classifiers (ICs). Some examples of ICs together withranking and tagging schemes may include: i) DescriptionComplexity—ranked from 1 being low complexity to 3 being highcomplexity; ii) Context Level—ranked from 1 being OS level, level 2being application, level 3 being functional, etc. to a being controlelement (e.g., button or slider); iii) Pre-requisite knowledge—being theID tag of other information sets in the Help Database; iv) SpecificTags—the tags provide information about a given help segment. Forexample: a) High Density Information (HDI)—a succinct set of text orgraphics containing a high amount of information (e.g., a tablecontaining a list of parameters and their usage); b) Additional Content(AC)—being additional material to be displayed if a user's cognitivestate is at a certain level, e.g., if the user shows either very high orlow understanding.

When help is requested, a set of help segments is determined. This maybe achieved by performing the following: if the user requests help on aspecific function, then this is directly related to specific helpsegments; if the requirement for help is triggered by a particularcognitive state for a given UI element (or sequence of UI elements) thenhelp is given for functionality associated with that UI element (orsequence of UI elements); if the user requests help but does not specifya function, then the recent viewing history of UI elements can beassessed and a set of help segments inferred; and for each case theresult is the OS has a list of relevant help segments, that also includeIC information.

In one example, if EEG data for a particular user indicates that theuser does not understand a feature (e.g., “Airplane Mode” above), andhelp is requested without specifying a specific topic, then thecomputing device implementing method 100 may provide a suggestion (e.g.,“Do you want help on ‘Airplane Mode’?”).

A Customization Algorithm then compares the cognitive and/or emotionalstates for given UI elements and compares these, and sequences of theseto threshold values in a lookup table, which then returns: the ICstogether with the tags and ranking that should be used to select helpsegments from a help database; and the order the help segments thatshould be displayed.

The help segments and order of display derived from the preceding stepare used to compile customized help information (e.g., a PHP) and thecustomized help information is then presented to the user.

Some of the embodiments discussed above have discussed customizing helpinformation based on a single user obtaining help for a single softwareprogram. The estimation of familiarity and understanding is made via EEGdata of the user alone within the context of a single softwareapplication. However, a number of alternatives are possible.

For example, EEG data may be compared among multiple users. This couldinclude the ability to compare either the particular user's EEGmeasurements, or the estimation of familiarity or understanding inferredfrom those EEG measurements, with other user's information. With such asystem the estimation of a user's familiarity or understanding couldpotentially be made with greater accuracy.

In some embodiments, the modification of text performed when customizinghelp information could be made in such a way that the particular user ismore likely to find it beneficial (e.g., once modifications are knownthat helped or did not help other users exhibiting similar patterns offamiliarity or understanding, it is more likely that a suitableconfiguration could be found for the user in question). In such systems,the method 100 could include determining that a given customization ofhelp information that describes how to use the UI was helpful to adifferent user that exhibited one of the one or more selected cognitivestates when that different user viewed the UI and/or interpreted helpinformation that describes how to use the UI. In such an embodiment, thecustomizing of block 106 could be further based on the givencustomization. Whether or not a customization was “helpful” could bedetermined, e.g., by asking a given user to provide a rating to the helpinformation recently provided to them.

The customization of help information may include EEG measurements madewhile the particular user is using other software programs, for exampleif two icons are each presented in a different software application, andare similar in appearance (as ascertained by an image comparisonalgorithm), yet the user seemingly has a low understanding of one and ahigh understanding of the other then it is more likely it is thecontextual use of that functionality that is causing the lack ofunderstanding. In such embodiments, the method 100 may include:determining one or more additional cognitive states that characterize astate of the particular user when the user views a different UI and/orthe help information that describes how to use the different UI; andcomparing the one or more selected cognitive states to the one or moreadditional cognitive states. In such embodiments, the customizing ofblock 106 could be further based on the one or more additional cognitivestates of the different UI.

Additional physiological measurements that can be correlated to a user'sunderstanding may also be taken in order to reduce the effects of signalnoise, and more accurately identify the cognitive state thatcharacterizes the particular user. This could include, for example, bodytemperature, blood pressure, heart rate, etc. Such additionalphysiological measurements could thereby improve the personalization ofhelp information by more accurately categorizing the state of the userat a given time.

As discussed above, in some embodiments help information may beorganized as “help segments,” each being (i) categorized by function,and where appropriate (ii) associated with a given UI element and (iii)ranked or tagged using a set of Information Classifiers (ICs). Helpsegments may include segments of text, images, tables etc. which whencombined form a coherent help page. Help segments may also include audioand/or video files that can be played back for a user as helpinformation. The audio and/or video files may be embedded on apersonalized help page (PHP), for example.

As discussed above, gaze tracking may be used to correlate EEG data witha particular UI element and/or with a particular section of help beingviewed. A UI element can include a graphic or object on a screen thatconveys some meaning and/or offer the option to carry out an action ororganize the displayed content. Examples of UI elements include icons,control objects (e.g., buttons volume sliders, On/Off sliders), and textsegments (e.g., “Airplane Mode” in FIG. 7 ). In some examples, UIelements may be overlapping (e.g., a first UI element which is a buttonmay be fully contained within a second UI element which is a panel). Inother embodiments, UI elements may be physical devices on a machine(e.g., buttons 46, knobs 47, dials 48, sliders 49, and switches 50 onmachine 42 in FIG. 5 ). Thus, the UI elements may be GUI elements (FIG.4 ), but may also be non-GUI, i.e., physical, elements (FIG. 5 ).

When correlating gaze tracking and EEG data, the notion of a “UI elementarea” may be used, which refers to a geometric area associated with agiven UI element or section of displayed help information. UI elementareas may be in the form of rectangles, circles, or any other such shapethat can be geometrically described. UI element areas may be used by theOS performing the correlating of gaze tracking data with EEG data.

In some examples, the customization of help information may be based ona sequence of viewed UI elements. These elements could be viewedsequentially by the user (e.g., as discussed in connection with FIG. 7), or could be viewed according to some sort of pattern (e.g., the usermay look at a button, then a section of text, then at the button again).

FIG. 8 illustrates an example computing device 300 configured forhelping a particular user use a UI by customizing help information basedon EEG data. The computing device includes a processor 302,communication interface circuit 304, and memory circuit 306. Theprocessor 302 includes one or more processor circuits, (e.g., one ormore microprocessors, microcontrollers, digital signal processors,application specific integrated circuit (ASIC), or the like) configuredwith appropriate software and/or firmware to carry out one or more ofthe techniques discussed above. The communication interface circuit 304obtains EEG data from a BCI (not shown) that measures the EEG data froma particular user through either a wireless interface, or a wiredinterface. If wireless communication is used, the communicationinterface circuit 304 may be configured according to one or more knownwireless communication standards (e.g., 3GPP or 802.11 standards). Ifwired communication is used, the communication interface circuit 304 mayobtain EEG data through an optional input port 316. The memory circuit306 may include one or several types of memory such as read-only memory(ROM), random-access memory, cache memory, flash memory devices, opticalstorage devices, etc.

The computing device 300 is configured to help a particular user use auser interface (UI). In particular, the processor 302 is configured toobtain EEG data that indicates brain activity of a particular userduring a period in which that user views a UI and/or interprets helpinformation that describes how to use the UI. The processor 302 isfurther configured to select, based on the EEG data and from amongmultiple predefined cognitive states, the one or more cognitive statesthat characterize the particular user during the period. The processor302 is further configured to assist the particular user to use the UI bybeing configured to customize the help information for the particularuser based on the one or more selected cognitive states.

The customized help information is displayed on an electronic display314 that is either in communication with the computing device 300 (e.g.,an external display) or is part of the computing device 300 (e.g., anembedded display). In some embodiments, the UI for which helpinformation is provided is a graphical user interface (GUI) presented onthe electronic display 314. In other embodiments, the UI is a UI for adevice (e.g., buttons, knobs, dials, switches, sliders, etc. as shown inFIG. 5 ) and the help information for that UI is provided on theelectronic display 314. The computing device 300 runs an operatingsystem such that, in some embodiments, information is presented to theparticular user in a graphical manner.

EEG data 308 is received through the communication interface circuit 304and may be stored in the memory circuit 306. In some embodiments, thecomputing device 300 includes a camera 318 operative to record images ofthe eyes of the particular user during a viewing period to obtain gazetracking data 310 which may be stored in memory circuit 306. The gazetracking data 310 may be correlated with the EEG data to determine withspecific UI elements and/or specific sections of help information towhich the EEG data corresponds. In some embodiments, the gaze trackingdata 310 is either received from another computing device, or is omittedentirely. Customizable help information for the UI in question is storedeither in memory circuit 306 or another memory device (e.g., as helpsegments in a help database).

Although a tablet computing device 36 is shown in FIG. 4 , it isunderstood that the computing device 300 of FIG. 8 may include anynumber of computing devices, such as smartphones, tables, cardashboards, home automation interfaces, laptop computers, desktopcomputers, etc.

In some embodiments, the computing device 300 includes a computerprogram product (CPP) 312 stored in a non-transitory computer-readablemedium (e.g., memory circuit 306) for helping a particular user use aUI. The computer program product includes software instructions which,when run by processor 302 of the computing device 300, configures thecomputing device 300 to: i) obtain electroencephalography (EEG) datathat indicates brain activity of a particular user during a period inwhich that user views a user interface (UI) and/or interprets helpinformation that describes how to use the UI; ii) based on the EEG data,select, from among multiple predefined cognitive states, the one or morecognitive states that characterize the particular user during theperiod; and iii) assist the particular user to use the UI by customizingthe help information for the particular user based on the one or moreselected cognitive states.

In embodiments where gaze tracking is used, there is an operating system(OS) running on the computing device 300, and the OS can determine sizeand location of various UI elements and/or sections of help information,and can determine the start and end time a UI element (or section ofhelp information) is displayed on the electronic display 314. In suchembodiments, if a camera 318 is included, and the OS knows the locationof the camera with respect to the electronic display 314 and a relativeangle of the electronic display 314 and camera 318 with respect to theparticular user, then the computing device 300 can record gaze trackingdata for a given user. In other embodiments, gaze tracking data may bereceived from another computing device. In the memory circuit 306, alookup table (e.g., Tables 1-2) may be stored that associates ERPsignals derived from a BCI with cognitive states and/or emotionalstates.

The various embodiments discussed above provide improvements over theprior art. Techniques are provided to customizing help information for aUI in a way that reflects the understanding of individual users. Thus,as a user's understanding of the UI changes over time, differentcustomizations can be performed. Also, in some embodiments, helpinformation can be customized based on an emotional state of the user,which can make interpreting that help information a more positiveexperience for the user.

The present disclosure may, of course, be carried out in other ways thanthose specifically set forth herein without departing from essentialcharacteristics of the present disclosure. The present embodiments areto be considered in all respects as illustrative and not restrictive,and all changes coming within the meaning and equivalency range of theappended claims are intended to be embraced therein.

1. A method for providing assistance to a user, the method comprising:obtaining electroencephalography (EEG) data that indicates brainactivity of the user during a period of time in which the user is usinga device; based on the EEG data, determining a first state of the user,wherein the first state is a cognitive state or an emotional state; andassisting the user to use the device or software running on the devicebased on the determined first state of the user.
 2. The method of claim1, wherein determining the first state comprises selecting, from amongmultiple predefined cognitive states, a cognitive state thatcharacterizes the user during the period of time, or determining thefirst state comprises selecting, from among multiple predefinedemotional states, an emotional state that characterizes the user duringthe period of time.
 3. The method of claim 1, wherein assisting the userbased on the determined first state comprises: customizing, based on thedetermined first state, help information that describes how to use thedevice and/or software; and outputting the customized help information.4. The method of claim 1, wherein during the period of time a userinterface (UI) is visible, and assisting the user comprises assistingthe user to use the UI based on the determined first state.
 5. Themethod of claim 1, wherein the first state is a first cognitive state,the method further comprises determining a first emotional state of theuser based on the EEG data, and assisting the user to use the device orsoftware based on the determined first state of the user comprisesassisting the user to use the device or software based on both the firstcognitive state and the first emotional state.
 6. The method of claim 1,further comprising: tracking the gaze of the user; and based on thetracked gaze of the user, associating the EEG data with one or moreelements of a user interface (UI) on the device or displayed using adisplay screen of the device.
 7. The method of claim 6, whereinassociating the EEG data with one or more elements of the UI based onthe tracked gaze of the user comprises determining, based on the trackedgaze of the user, one or more individual elements of the UI that theuser is viewing and associating the EEG data with the one or moreindividual elements of the UI, and assisting the user comprisesassisting the user to use the UI based on the one or more individualelements of the UI.
 8. The method of claim 1, wherein the EEG datacomprises event related potentials (ERPs) corresponding to individualelements of a user interface (UI) on the device or displayed using adisplay screen of the device, determining the first state comprises,based on the ERPs, selecting, from among multiple predefined states, astate that characterizes the user during the period of time.
 9. Themethod of claim 1, wherein assisting the user to use the device orsoftware based on the determined first state of the user comprisesdetermining whether the determined first state indicates a lack ofsemantic understanding.
 10. A computing device for providing assistanceto a user, the computing device comprising: memory; and processingcircuitry coupled to the memory, wherein the computing device isconfigured to: obtain electroencephalography (EEG) data that indicatesbrain activity of the user during a period of time in which the user isusing a device; based on the EEG data, determine a first state of theuser, wherein the first state is a cognitive state or an emotionalstate; and assist the user to use the device or software running on thedevice based on the determined first state of the user.
 11. Thecomputing device of claim 10, wherein assisting the user to use thedevice or software based on the determined first state of the usercomprises determining whether the determined first state indicates alack of semantic understanding.
 12. The computing device of claim 10,wherein determining the first state comprises selecting, from amongmultiple predefined cognitive states, a cognitive state thatcharacterizes the user during the period of time, or determining thefirst state comprises selecting, from among multiple predefinedemotional states, an emotional state that characterizes the user duringthe period of time.
 13. The computing device of claim 10, whereinassisting the user based on the determined first state comprises:customizing, based on the determined first state, help information thatdescribes how to use the device and/or software; and outputting thecustomized help information.
 14. The computing device of claim 10,wherein during the period of time a user interface (UI) is visible, andassisting the user comprises assisting the user to use the UI based onthe determined first state.
 15. The computing device of claim 10,wherein the first state is a first cognitive state, the computing deviceis further configured to determine a first emotional state of the userbased on the EEG data, and assisting the user to use the device orsoftware based on the determined first state of the user comprisesassisting the user to use the device or software based on both the firstcognitive state and the first emotional state.
 16. The computing deviceof claim 10, wherein the computing device is further configured to:track the gaze of the user; and based on the tracked gaze of the user,associate the EEG data with one or more elements of a user interface(UI) on the device or displayed using a display screen of the device.17. The computing device of claim 16, wherein associating the EEG datawith one or more elements of the UI based on the tracked gaze of theuser comprises determining, based on the tracked gaze of the user, oneor more individual elements of the UI that the user is viewing andassociating the EEG data with the one or more individual elements of theUI, and assisting the user comprises assisting the user to use the UIbased on the one or more individual elements of the UI.
 18. Thecomputing device of claim 10, wherein the EEG data comprises eventrelated potentials (ERPs) corresponding to individual elements of a userinterface (UI) on the device or displayed using a display screen of thedevice, determining the first state comprises, based on the ERPs,selecting, from among multiple predefined states, a state thatcharacterizes the user during the period of time.
 19. The computingdevice of claim 10, wherein the device is the computing device.
 20. Acomputer program product comprising a non-transitory computer readablemedium storing instructions for configuring a computing device to:obtain electroencephalography (EEG) data that indicates brain activityof the user during a period of time in which the user is using a device;based on the EEG data, determine a first state of the user, wherein thefirst state is a cognitive state or an emotional state; and assist theuser to use the device or software running on the device based on thedetermined first state of the user.